Field of the Invention
The present invention relates to a technique for recognizing a user operation based on a captured image.
Description of the Related Art
In recent years, there is a situation where in an environment, such as augmented reality (AR) or mixed reality (MR), a touch operation is performed on an operation surface that does not physically exist (a virtual surface). Further, there is a situation where a user interface (hereinafter, a “UI”) is projected on any operation surface, such as a wall or a desk, and a touch operation is performed on the projected UI. In such situations, it is often the case that the distance between the operation surface and a part of a predetermined operation object (e.g., the fingertip of a user) is determined using an image acquisition means, such as a stereo camera, a range image sensor, or the like, and based on the determination of the distance, it is determined whether the operation surface and the part of a predetermined operation object are in contact with each other. Specifically, a predetermined threshold is provided for the distance between the operation surface and the fingertip of the user.
In a case where an operation based on the movement of a part of a body is recognized using an image, depending on the positional relationship between an image acquisition means and an operator, a part of the body of the operator to be detected may enter the blind spot of the image acquisition means, and there arises an issue that the movement of the part of the body may not be able to be correctly recognized.
In response to the above issue, a technique discussed in the publication of Japanese Patent Application Laid-Open No. 2013-205983 continuously tracks the position and the orientation of the end of the hand of an operator, using a three-dimensional model of the end of the hand. As a result, even if the end of the hand enters a blind spot, it is possible to roughly estimate the position of the end of the hand and continuously recognize a spatial gesture operation based on the movement of the end of the hand.
As in the publication of Japanese Patent Application Laid-Open No. 2013-205983, if the position of a portion hidden in a blind spot is estimated, it is possible to complement information necessary for recognition of a spatial gesture operation based on a large movement of a hand, or the like. It is, however, difficult to accurately estimate the coordinates of an indicated position which is indicated by a fingertip hidden in a blind spot, and to determine whether the fingertip is touching an operation surface.
When a multi-touch operation in which movements of two instruction positions are cooperated is performed to issue an instruction to enlarge or reduce or to rotate a displayed image, only either one of the two indicated positions may be hidden in a blind spot. In the multi-touch operation, if the acquisition of the coordinates of either one of the indicated positions or the touch determination of either one of the indicated positions is failed, it is difficult to correctly recognize the content of the operation.